An End-to-End Framework Based on Vision-Language Fusion for Remote Sensing Cross-Modal Text-Image Retrieval

Remote sensing cross-modal text-image retrieval (RSCTIR) has recently attracted extensive attention due to its advantages of fast extraction remote image information and flexible human–computer interaction. Traditional RSCTIR methods mainly focus on improving the performance uni-modal feature separately, most rely pre-trained object detectors obtain better local representation, which not only lack multi-modal interaction information, but also cause training gap between detector task. In this paper, we propose an end-to-end framework based vision-language fusion (EnVLF) consisting two (vision language) encoders a muti-modal encoder can be optimized by multitask training. Specifically, achieve process, introduce vision transformer module for features instead detector. By semantic alignment visual text features, achieves same as features. addition, trained improve top-one top-five ranking performances after processing. Experiments common RSICD RSITMD datasets demonstrate that our EnVLF state-of-the-art performance.